External IR illuminator enabling improved head tracking and surface reconstruction for virtual reality

ABSTRACT

Disclosed embodiments include methods and systems for utilizing a structured projection pattern to perform depth detection. In some instances, the structured projection pattern forms a dot pattern, which is projected by a projector, wherein the projector includes one or more infrared (IR) light dot pattern illuminators for projecting an IR light dot pattern to a surrounding environment. The IR dot pattern light that is at least partially reflected off one or more objects in the surrounding environment is detected by one or more cameras attached to a head-mounted display (HMD). The HMD, which is physically untethered from the projector, utilizes the captured IR dot pattern light reflections to track movement of the HMD and/or perform depth detection of one or more objects in the environment surrounding the HMD.

BACKGROUND

Mixed-reality systems, including virtual-reality and augmented-realitysystems, have received significant attention because of their ability tocreate truly unique experiences for their users. For reference,conventional virtual-reality (VR) systems create a completely immersiveexperience by restricting their users' views to only a virtualenvironment. This is often achieved through the use of a head-mounteddevice (HMD) that completely blocks any view of the real world. As aresult, a user is entirely immersed within the virtual environment. Incontrast, conventional augmented-reality (AR) systems create anaugmented-reality experience by visually presenting virtual objects thatare placed in or that interact with the real world.

As used herein, VR and AR systems are described and referencedinterchangeably. Unless stated otherwise, the descriptions herein applyequally to all types of mixed-reality systems, which (as detailed above)includes AR systems, VR reality systems, and/or any other similar systemcapable of displaying virtual objects.

The disclosed mixed-reality systems use one or more on-body devices(e.g., the HMD, a handheld device, etc.). The HMD provides a displaythat enables a user to view overlapping and/or integrated visualinformation in whatever environment the user is in, be it a VRenvironment or an AR environment. By way of example, as shown in FIG. 1,a mixed-reality system may present virtual content to a user in the formof a simulated vase resting on a real table surface.

Continued advances in hardware capabilities and rendering technologieshave greatly improved how mixed-reality systems render virtual objects.However, the process of immersing a user into a mixed-realityenvironment creates many challenges, difficulties, and costs,particularly with regard to determining three-dimensional spatialinformation around the user and tracking a user's movement so the visualdisplay of information can be correctly presented to the user.

For instance, by way of example, conventional passive stereo depthdetection systems fail to adequately determine the depth of a smooth orlow texture surface (e.g., a wall) in a mixed-reality environmentbecause those systems fail to adequately distinguish one part of thesmooth/textureless surface from another part. As such, there is asubstantial need to improve how depth is detected, especially forsmooth/textureless surfaced objects in mixed-reality environments.

Additionally, many conventional HMD systems require separate/additionalhardware that is mounted to the HMD for performing depth detection, fromthe hardware that is required to perform head tracking. This additionalhardware adds to the overall cost, weight, battery consumption and sizeof the HMD systems, and leads to resource allocation issues on HMDsystems.

Another problem with conventional HMD systems is that they often performtracking and depth detection poorly in low light environments, due tothe lack of light even when the HMD is configured to emit light.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is provided only toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY

Disclosed embodiments include methods and systems for utilizing astructured projection pattern to perform depth detection. In someinstances, the structured projection pattern forms a dot pattern, whichis projected by a projector, wherein the projector includes one or moreinfrared (IR) light dot pattern illuminators for projecting an IR lightdot pattern to a surrounding environment. Embodiments disclosed hereinare also able to operate using visible light to detect depth. As aresult, visible light, IR light, or a combination of both visible lightand IR light may be used to determine an object's depth. The IR dotpattern light that is at least partially reflected off one or moreobjects in the surrounding environment is detected by one or morecameras attached to a head-mounted display (HMD). These cameras are alsoable to detect visible light. The HMD, which is physically untetheredfrom the projector, utilizes the captured IR dot pattern lightreflections and/or the visible light reflections to track movement ofthe HMD and/or perform depth detection of one or more objects in theenvironment surrounding the HMD.

In some embodiments, the projector projects light to provide additionallighting and/or texture to an environment surrounding a HMD to improveinside-out tracking (i.e., tracking based on cameras in the HMD).

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the invention may be realized and obtained by means of theinstruments and combinations particularly pointed out in the appendedclaims. Features of the present invention will become more fullyapparent from the following description and appended claims, or may belearned by the practice of the invention as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features can be obtained, a more particular descriptionof the subject matter briefly described above will be rendered byreference to specific embodiments which are illustrated in the appendeddrawings. Understanding that these drawings depict only typicalembodiments and are not therefore to be considered to be limiting inscope, embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 shows a head-mounted device (HMD) structured to identify itslocation and orientation with respect to its surrounding environment(i.e. motion detection) as well as to determine the depth of an objectin that environment. FIG. 1 also illustrates a table and a virtualobject (i.e. the vase) that are visible to the user of the HMD.

FIG. 2 illustrates a HMD that includes a stereo camera pair which can beused to perform motion detection and which can also be used to performdepth detection using an overlapping field of view region existingbetween the two cameras' fields of view.

FIG. 3 shows an example environment in which the HMD may be used, andthis example environment includes some objects that havetextureless/smooth surfaces.

FIG. 4 demonstrates a projector projecting an IR dot patternillumination into a HMD's surrounding environment

FIG. 5 shows a projector projecting an IR dot pattern illumination intoa HMD's surrounding environment based at least in part oncharacteristics (e.g., pose and/or motion) of the HMD.

FIG. 6 illustrates a projector projecting an IR flood illumination intoa HMD's surrounding environment.

FIG. 7 demonstrates a projector projecting an IR flood illumination intoa HMD's surrounding environment, with attention drawn to anchor pointsto be utilized by the HMD to track motion of the HMD.

FIG. 8 portrays an example in which more than one projector works inconcert to project IR light into an environment surrounding a HMD.

FIG. 9 shows an example of head tracking and depth detector systemcomponents which components may be used to perform head tracking anddepth detection using visible light and/or IR light.

FIG. 10 provides a flow diagram representing a method for performingdepth detection with a HMD and projector which are physically untetheredfrom one another.

FIG. 11 provides flow diagram representing a method for trackingmovement of a HMD with the HMD and a projector physically untetheredfrom the HMD. The HMD is able to track the movements using visiblelight, IR light, or a combination of visible and IR light.

FIG. 12 illustrates an example computer system that may be used toperform embodiments disclosed herein.

DETAILED DESCRIPTION

At least some of the embodiments described herein relate to head-mounteddevices (HMDs) configured to perform depth detection by generating a 3Dgeometry mapping of the surrounding environment. As an initial matter,the HMD may include one or more cameras, including a stereo camera paircomprising a first and second camera. Both cameras are mounted on theHMD, and are collectively and/or independently able to detect visiblelight and infrared (IR) light. Such positioning is beneficial for motiontracking purposes because it allows the cameras to capture a largeamount of area in the surrounding environment. By capturing more area,the HMD is better able to track movements. Furthermore, in at least someimplementations, at least a part of the cameras' fields of view overlapwith one another (i.e. an overlapping field of view region).

At least some of the embodiments described herein relate to a projectorincluding one or more IR light illuminators. The projector is configuredto emit an IR structured illumination pattern that spans an illuminationarea (e.g., an IR dot pattern and/or an IR flood illumination). Theprojector is physically untethered from the HMD and is aimed/oriented insuch a particular manner that the IR structured illumination patternprojected from the projector at least partially overlaps with the fieldsof view of one or both cameras of the HMD and preferably both fields ofview. Such a configuration is beneficial for enabling the camera systemto capture the reflections of the IR structured illumination pattern aswell as visible light that reflects off of the same object(s) in thesurrounding environment by both cameras of the camera system.

By obtaining digital image content corresponding to the IR dot-patterntexturing and/or visible light, the HMD is able to improve the qualityof the calculated 3D geometry of the surrounding environment bydetermining the stereoscopic depth(s) for each object. This beneficiallyimproves the manner in which objects, particularly textureless/smoothsurfaced objects, are reconstructed with depth detection processing byHMD systems.

Inclusively or additionally, the IR structured illumination patternlight projected by the projector can also be utilized by one or morecameras of the HMD to improve the tracking processing performed by HMDby facilitating the manner in which anchor points are identified (withreflected pattern light and/or visible light) within the surroundingenvironment.

Accordingly, it will be appreciated that the disclosed embodiments canbe used to provide significant improvements over how HMDs perform depthdetection, particularly for objects having textureless/smooth surfaces.This is accomplished, for example, by using the HMD to apply and senseIR dot-pattern texturing applied to the surfaces of the objects in theenvironment surrounding the HMD as well as by sensing reflected visiblelight. This IR dot-pattern beneficially augments any detected visiblelight when performing depth detection. In this manner, it is possible toclearly and accurately determine any object's depth, even when thatobject has textureless/smooth surfaces.

Additionally, in at least some instances, the present embodimentsimprove the tracking of the HMD by providing techniques for identifyingadditional anchor points that may not be identifiable withoutsupplemental illumination from a projector. Furthermore, because theprojector is physically untethered from the HMD, the systems and methodsdescribed herein may reduce or eliminate resource allocation issues insystems performing depth detection and head tracking (e.g., batteryconsumption, overall weight, etc.).

Having just described some of the various high-level features andbenefits of the disclosed embodiments, attention will now be directed toFIGS. 1 through 11. These figures illustrate various architectures,methods, and supporting illustrations related to utilizing a projectorto add visible texture to a surface to better determine that surface'sdepth and/or improve head tracking. Subsequently, the disclosure willturn to FIG. 12, which presents an example computer system that may beused to facilitate the disclosed principles.

Improved Depth Detection and Head Tracking Systems

Attention is now directed to FIG. 1, which illustrates an exampleenvironment 100 of a user 105 using a HMD 110. The HMD 110 is an exampleof a mixed-reality system that is able to render virtual content for theuser 105. As previously noted, the HMD 110 may be a VR system or an ARsystem, such that environment 100 may be a VR environment or an ARenvironment. The term environment, mixed-reality environment andsurrounding environment will be used interchangeably herein to refer toenvironment 100 and other HMD environments referenced herein.

In world-locked holograms/mixed-reality environments (akaworld-stabilized imaging), a user may experience discomfort when his/herhead movement is not matched to what is visually displayed. Therefore,it is desirable to provide the user 105 with as pleasant an experienceas possible while the user 105 is wearing the HMD 110 by determining theuser's position in relation to the various objects in the environment100 (i.e. to perform depth detection and head tracking).

In FIG. 1, the environment 100 is shown as including a first object 115and a second object 120. To obtain an accurate mapping of the realobjects in the scene (aka mixed-reality environment), it is beneficialto know how far away these objects are from the user 105 at any givenmoment. By following the principles disclosed herein, significantadvantages are realized because highly accurate depth determinations maybe performed. By performing these depth determinations, themixed-reality environment, which is created by the HMD 110, canaccurately place virtual objects that interact with the real world. Thisresults in a more life-like interaction of virtual and real worldobjects, and the user 105's experience will be significantly improved.

FIG. 2 shows a HMD 200 that is specially configured to perform advanceddepth determinations in addition to rendering mixed-realityenvironments. For reference, this HMD 200 is one example implementationof the HMD 110 from FIG. 1. FIG. 2 is illustrated from a topperspective, looking down at the HMD 200, as indicated by the “x, y, z”direction legend.

As shown, HMD 200 includes a head-tracking stereo camera pair whichincludes at least two cameras, namely camera 205 and camera 210, both ofwhich are mounted on the HMD 200. According to the disclosedembodiments, the head tracking stereo camera pair may be used formultiple different operations, including, but not limited to, capturingimages for tracking the movements of the HMD 200, as well as capturingimages for determining depth.

Although HMD 200 is shown as including only two cameras, the HMD 200 mayactually include any number of cameras. For instance, the HMD 200 mayinclude 3 cameras, 4 cameras or more than four cameras. As such, the HMD200 is not limited only to two cameras.

Camera 205 is shown as including an optical axis 215. For reference, acamera's optical axis is an imaginary “line” that passes through thedirect center of the camera's lens. As a practical example, an opticalaxis is akin to the point where the camera is being aimed. In additionto the optical axis 215, FIG. 2 also shows that camera 205 has a fieldof view 220. In some implementations, camera 205 includes a wide-anglelens such that the field of view 220 is also a wide-angle field of view.This wide-angle field of view may span a range anywhere from 45 degreesup to 180 degrees horizontally (in ultra-wide-angle cameras) andanywhere from 45 degrees up to 120 degrees vertically.

Camera 210 may be configured similarly to camera 205. For instance,camera 210 similarly includes an optical axis 225 and a field of view230. By combining the fields of view of the two cameras, a very largespanning area (e.g., 170 degrees, 180 degrees, etc.) around the HMD maybe captured.

These cameras may be configured in many different ways. For example, insome implementations, both of the cameras 205 and 210 are configured asglobal shutter cameras. In other implementations, however, the cameras205 and 210 are configured as rolling shutter cameras. Of course,combinations of global shutter and rolling shutter cameras may also beused. As an example, the camera 205 may be a global shutter camera whilethe camera 210 may be a rolling shutter camera. In a preferredembodiment, a global shutter camera is used because rolling shuttercameras are more prone to motion blur. Of course, the HMD 200 may havemany cameras, some of which are global shutter and some of which arerolling shutter.

In some implementations, the cameras 205 and 210 (and in particular thepixels of these cameras) may be configured to detect, or rather besensitive to, different spectrums of light (e.g., visible light andinfrared (IR) light). For reference, the visible light spectrum rangesanywhere from around 380 nanometers (nm) up to and including about 740nm. More specifically, violet light ranges from 380 nm to 435 nm. Bluelight ranges from 435 nm to 500 nm. Green light ranges from 500 nm to520 nm. Yellow light ranges from 565 nm to 590 nm. Red light ranges from625 nm to 740 nm.

To capture light (e.g., visible light and/or IR light), pixels of thecameras 205 and 210 are exposed to light during a pre-selected exposureperiod. During this exposure period, the pixels sense photons andgenerate an electrical response based on the amount and/or intensity ofthe sensed photons. This exposure period may be a variable exposure timesuch that it changes from one exposure instance to another exposureinstance. As will be described in more detail later, the HMD is able tocontrol the exposure time of the cameras 205 and 210 to coincide/syncwith one or more other light emitting components (e.g., an IRdot-pattern illuminator and/or a flood IR illuminator).

In contrast to visible light, infrared (IR) light is invisible to ahuman's eye and has a wavelength that is longer than the wavelengths forvisible light. The infrared light spectrum starts at the trailing edgeof the red light spectrum, around 700 nm, and extends to at least 1 umin length.

With that said, cameras 205 and 210 (at a pixel level) are configured todetect both visible light and IR light. In some instances, one or moreof the cameras 205 and 210 are monochromatic cameras (i.e. greyscale).In some instances, one or more of the cameras 205 and 210 are chromaticcameras.

Of course, the cameras 205 and 210 may also be configured to detect onlyportions of the visible light spectrum and portions of the IR lightspectrum. This may be achieved through the use of one or more opticalbandpass filters in the lens. For brevity, the remaining disclosure willsimply use the singular form of the term bandpass filter even thougheach camera may be configured with its own similarly configured oruniquely different bandpass filter.

The bandpass filter is configured, in some instances, to allow only aselected range of visible light to pass through and be detected by oneor more corresponding camera(s) and while also allowing some or all IRlight to also be detected by the same camera(s). Additionally, oralternatively, the bandpass filter may be configured to allow only aselected range of IR light to pass through and be detected by the one ormore corresponding camera(s) while allowing some or all visible light topass through and be detected by the same camera(s).

By way of example, the bandpass filter is configured in some embodimentsto pass visible light having wavelengths between approximately 400 nm upto approximately 700 nm. In some embodiments, the bandpass filter isalso specifically configured to pass IR light having wavelengthscorresponding to the same wavelengths of IR light emitted by an IR lasermounted on the HMD 200 (to be discussed in more detail later). Oneexample of the IR laser's wavelength may be approximately 850 nm. Assuch, the bandpass filter may pass IR light having wavelengths within athreshold value of the IR laser's wavelengths (e.g., within 10 nm, 20nm, 30 nm, 40 nm, 50 nm, etc. of the emitted IR wavelength) while notpassing other IR light wavelengths.

In view of the foregoing, it will be appreciated that one or bothcameras 205 and 210 may include a bandpass filter that allows at leastsome visible light to pass through the bandpass filter (whilepotentially filtering out some visible light) and at least some IR lightto pass through the bandpass filter (while potentially filtering outsome IR light). Likewise, in some implementations, camera 205 and/orcamera 210 may also omit any IR light filter.

FIG. 2 also shows how the cameras 205 and 210 may be positioned inrelation to each other on the HMD 200. For example, at least a part ofthe field of view 220 of camera 205 is shown as overlapping at least apart of the field of view 230 of camera 210 thus forming the overlappingregion 235 (aka an “overlapping field of view region”). This overlappingregion 235 is beneficial for a number of reasons, which will bediscussed later.

However, it will be appreciated that certain advantages of theembodiments disclosed herein may be realized by HMD systems with asingle camera or with multiple cameras with fields of view that do notoverlap. For example, a HMD with two cameras with non-overlapping fieldsof view will benefit from additional anchor points which may be utilizedin tracking the movement of the HMD, even though two cameras without anoverlapping field of view cannot perform stereoscopic depthcalculations.

In some configurations, the cameras may be horizontally offset (e.g.,offset relative to a horizontal alignment of the HMD 200 in they-direction plane). For instance, camera 205 may be pointed slightlydownward or upward in the y-direction while camera 210 may be alignedwith the horizontal plane (e.g., y-direction). In this manner, thecamera 205 may have a y-angle offset in relation to the horizontalalignment of the HMD 200. Relatedly, the camera 210 may be pointedslightly downward or upward in the y-direction relative to camera 205,while camera 205 is aligned with the y-direction horizontal plane. Ofcourse, combinations of the above are also available. For instance,camera 205 may be pointed slightly downward relative to the horizontalplane and camera 210 may be pointed slightly upward relative to thehorizontal plane, and vice versa. Alternatively, cameras 205 and 210 arehorizontally aligned, such that they do not have any y-angle offset andsuch that they are pointed directionally level in the y-direction.

Additionally, or alternatively, to the above horizontalalignments/offsets, cameras 205 and 210 may also be aligned/offset inother directions. For instance, FIG. 2 shows that the optical axis 215of camera 205 is angled (i.e. non-parallel) in relation to the opticalaxis 225 of camera 210 in the x-direction. Such a configuration issometimes beneficial because it allows the cameras 205 and 210 tocapture a larger area of the surrounding environment, thus providingmore reference area when performing movement detection (e.g., headtracking). This angle offset may be any selected angle. Example anglesinclude, but are not limited to 5 degrees, 10, 15, 20, 25, 30, 35, 40,45, 50, 55, 60, 65, 70, 75, 80, 85 degrees, and so on.

Although FIG. 2 and the remaining figures show the cameras 205 and 210angled in relation to one another, the embodiments should not be limitedto such a configuration. In fact, in some instances, the optical axes215 and 225 are aligned in parallel with one another in the x direction.In any event, and regardless of which orientation is used, the disclosedembodiments advantageously create overlapping region 235 with the fieldsof view 220 and 230.

Yet another configuration is available for the cameras 205 and 210. Toillustrate, the vertical positions of the cameras 205 and 210 (i.e. therelative height of the cameras along the y-direction on the HMD 200) mayalso vary. As an example, camera 205 may be positioned below camera 210on the HMD 200. Alternatively, camera 210 may be positioned below camera205 on the HMD 200. Otherwise, the cameras 205 and 210 are mounted atthe same relative height/vertical position on the HMD 200. Accordingly,from this disclosure, it is clear that the positions and orientations ofthe cameras 205 and 210 may vary widely.

Now that the configurations for the cameras 205 and 210 have beenintroduced, the disclosure will turn to how these cameras 205 and 210may operate. Recall, the stereo camera system/pair (i.e. cameras 205 and210) are configured to detect light for performing movement detection(e.g., head tracking, hand tracking, object tracking, etc.), as well asdepth detection. With regard to head tracking, the stereo camera pairactually constitutes an “inside-out” head tracking system because thestereo camera pair is mounted on the HMD 200.

An “inside-out” head tracking system tracks the position of a HMD (e.g.,HMD 200) by monitoring the HMD's position in relation to its surroundingenvironment. This is accomplished through the use of tracking cameras(e.g., cameras 205 and 210) that are mounted on the HMD itself and thatare pointed away from the HMD. In contrast, an “outside-in” trackingsystem uses cameras or external light illuminators that are mounted inthe environment and that are pointed toward the HMD. In this manner,inside-out head tracking systems are distinguished from outside-in headtracking systems.

As shown, cameras 205 and 210 are mounted on the HMD 200 (i.e. theobject being tracked) and may be (but are not required to be) slightlyoriented away from each other (as shown by the angled orientation of theoptical axes 215 and 225). Stated differently, the optical axis 215 isangled in relation to the optical axis 225.

To capture as much of the surrounding environment as possible, camera205 and camera 210 may be positioned apart, at a preselected distancefrom each other, and may be angled away from each other. Thispreselected distance is referred to as a “baseline,” and it may be anydistance. Commonly, however, the baseline will range anywhere between atleast 4 centimeters (cm) up to and including 16 cm (e.g., 4.0 cm, 4.1cm, 4.2 cm, 4.5 cm, 5.0 cm, 5.5 cm, 6.0 cm, 6.5 cm, 7.0 cm, 7.5 cm, 8.0cm, 8.5 cm, 9.0 cm, 9.5 cm, 10.0 cm, 10.5 cm, 11.0 cm, 11.5, cm, 12.0cm, 12.5 cm, 13.0 cm, 13.5 cm, 14.0 cm, 14.5 cm, 15.0 cm, 15.5 cm, 16.0cm, or more than 16.0 cm or less than 4.0 cm). Often, the baseline is atleast 10 centimeters. Sometimes, the baseline is chosen to match themost common interpupil distance for humans, which is typically between5.8 cm and 7.2 cm. In general, a wider baseline allows for accuratedepth from stereo for an increased distance over narrower baselinedesigns. Other factors that may influence the accuracy of the camerasystem are the cameras' fields of view and their image resolution.

With the foregoing configuration, the stereo camera system is enabled tocapture a large area of the surrounding environment, thus enabling theHMD 200 to interpolate its own position in relation to that environment.In addition to performing head tracking, the HMD 200 (and specificallythe stereo camera pair along with the stereo camera pair's logicalcomponents) may be re-purposed, or rather multi-purposed, to alsoperform an improved form of depth detection. By re-purposing existinghardware components, the embodiments significantly reduce the cost forperforming depth detection, especially when compared to time-of-flightdepth detection systems.

As an initial matter, it is noted that humans are able to perceive“depth” because humans have a pair of eyes that work in tandem. Whenboth eyes are focused on an object, signals from the eyes aretransmitted to the brain. The brain is then able to interpolate depthusing any disparity existing between the information captured from thetwo eyes.

Similar to how a human's eyes “focus” on an object when determiningdepth, the HMD 200 also obtains “focused” digital image content todetermine depth. Here, the “focused” digital image content is obtainedfrom camera images that include content corresponding to the overlappingregion 235 (i.e. camera 205's image and camera 210's image, both ofwhich include digital content corresponding to the overlapping region235). In this manner, the cameras 205 and 210 obtain separate images,but these images still have at least some similar content.

Here, an example will be helpful. Suppose a table was located in the HMD200's environment and that the HMD 200 was positioned so that the tablewas located within the overlapping region 235. In this scenario, cameras205 and 210 are each able to obtain a digital image that includesdigital content corresponding to the table. Consequently, at least someof the pixels in the image obtained by camera 205 will correspond to atleast some of the pixels in the image obtained by camera 210.Specifically, these “corresponding pixels” (i.e. the pixels in the oneimage that correspond to the pixels in the other image) are associatedwith the table.

Once these digital images are obtained, then the HMD 200 performscertain transformations (also called “re-projections”) on those digitalimages. These transformations correct for lens distortion and othercamera artifacts. Furthermore, the stereo images are re-projected onto avirtual stereo rig where both image planes lie inside a plane that isparallel to the stereo cameras' baseline. After re-projection,corresponding pixels are guaranteed to lie on the same horizontalscanline in left and right images. As a result, two “re-projected”images are formed, one for the image that was obtained by the camera 205and one for the image that was obtained by the camera 210. Any pixelsthat are similar/correspond between the two re-projected images now lieon the same horizontal plain.

After the re-projected images are created, the HMD 200 measures anypixel disparity that exists between each of the corresponding pixels inthe two images. Because the HMD 200 understands that the correspondingpixels in the two re-projected images are now in the same horizontalplain, the HMD 200 identifies that the disparity between thesecorresponding pixels corresponds (i.e. is proportional) with a depthmeasurement. Using this disparity, the HMD 200 assigns a depth value toeach pixel, thus generating a depth map for any objects located in theoverlapping region 235. Accordingly, the HMD 200, through the use of itsmulti-purposed head-tracking stereo camera pair, is able to perform bothmovement detection as well as depth detection.

The remaining portion of this disclosure uses many examples of camerasand head tracking stereo camera pairs (or simply stereo camera pairs).Unless stated otherwise, these cameras may be configured with any of thepositional/alignment configurations discussed above. Therefore,regardless of whether the system is performing head tracking and/ordepth detection, any of the cameras mentioned above, operating in any ofthe configurations mentioned above, may be used.

With that understanding, attention will now be directed to FIG. 3. Inthis illustration, an example environment 300 is provided, which may bepresented to a user (e.g., user 105 from FIG. 1) who is using a HMD(e.g., HMD 110 from FIG. 1 or HMD 200 from FIG. 2) to visualize amixed-reality environment.

Environment 300 includes a number of different features and objects. Forexample, environment 300 includes a textureless/smooth table top 305, atextureless/smooth wall 310, and a textured door frame 315, just to namea few. Of course, this is just one example of what an environment maylook like, and thus should not be considered limiting or otherwisebinding.

One problem that conventional depth perception systems have faced isdetermining depth for “textureless/smooth” objects (e.g., thetextureless/smooth table top 305 and the textureless/smooth wall 310).For textured surfaces, like the textured door frame 315, traditionaldepth detection systems are usually able to capture enough details toperform the stereo matching between the left and right cameras toadequately gauge the depth of those textured objects. Unfortunately,however, traditional depth detection systems are very inadequate indetermining the depth of textureless/smooth objects. In particular,traditional depth detection systems cannot collect enough information toadequately distinguish one part of the textureless/smooth object fromanother part, which may be further away.

For instance, if a user were to stand near the textureless/smooth tabletop 305, portions of the textureless/smooth wall 310 will besignificantly closer than other portions of the textureless/smooth wall310. However, traditional systems are unable to account for this changein depth because of a lack of texture on the surfaces and, hence a lackof reflected light that is used to determine the depth. As a result,traditional systems will often generate a false or otherwise misleadingdepth map for textureless/smooth objects like textureless/smooth wall310. If any virtual content is dependent on that false depth map, thenclearly the mixed-reality environment will be skewed and thus the user'sexperience will be hampered.

To address the above problems, some of the disclosed embodimentsbeneficially project, or rather add, texture to the environment. In someimplementations, this texture is in the form of an infrared (IR)dot-pattern illumination. Because the HMD's stereo camera pair (e.g.,camera 205 and 210 from FIG. 2) is sensitive to both visible andinfrared (IR) light, the stereo camera pair is able to detect the addedtexture and compute proper depth for any kind of object, even atextureless/smooth object. The HMD is thereby provided a picture withstructured light, thus improving depth quality determinations. Otherpossible implementations could combine visible light dot-patternprojectors with cameras that only capture visible light or IR dotprojectors with cameras that only capture IR light.

Attention is now directed to FIG. 4. In this illustration, an IRprojection pattern illuminator 405 projects/disperses IR light as an IRdot-pattern illumination 410 into the surrounding environment 400 of aHMD 403 in order to project, or rather add “texture” to object surfacesin the surrounding environment 400. This IR dot-pattern illumination 410may be projected to any predetermined illumination area within the HMD403 surrounding environment 400 (e.g., any area in the environment 300from FIG. 3). By way of non-limiting example, the IR projection patternilluminator 405 may be configured to project/disperse IR light in

$\frac{\pi}{2},\pi,\frac{3\pi}{2},{2\pi},\frac{5\pi}{2},{3\pi},\frac{7\pi}{2},$or 4π steradians, in any direction into the environment 400.

Although, FIG. 4 only shows a single IR projection pattern illuminator405 being used to project the IR dot-pattern illumination 410, it willbe appreciated that two or more IR projection pattern illuminators maybe utilized in at least some of the disclosed embodiments, (see FIG. 8),which are physically untethered from the HMD 403. It will also beappreciated that an IR projection pattern illuminator 405 may alsoinclude any combination of IR light emitting diode (LED), LED array, IRlaser diode, incandescent discharge illuminator, vertical-cavitysurface-emitting laser (VCSEL) and/or plasma discharge illuminator.

The IR dot-pattern illumination 405 may be generated in various ways.For instance, in a preferred embodiment, the IR dot-pattern illumination405 is generated using a diffraction limited laser beam, a collimatingoptic, and a diffractive optical element (DOE) (aka an opticaldiffuser). As such, the IR dot-pattern illuminator 400 may also includea collimating optic and a DOE to provide the desiredprojection/dispersion of the IR dot-pattern illumination 405. When an IRlaser shoots a diffraction limited laser beam of IR light into the DOE,then the DOE disperses the IR light in such a manner so as to projectthe pre-configured dot pattern illumination. Other IR LED, incandescentdischarge illuminator, VCSEL, plasma discharge illuminator, etc., may beused with more traditional imaging and re-projection techniques as well.

In an alternative embodiment, an etched lens may also be placed over topof an IR optical source/illuminator. In a first example, individual dotsmay be etched onto the lens to create the dot pattern. When thedot-pattern illuminator 400's IR laser emits a beam of IR light throughthis type of lens, the IR light unimpededly passes through the lens inthe areas that were not etched. However, for the dot areas that wereetched, the IR light may be impeded in accordance with the etchedpattern, thus projecting a dot pattern into the surrounding environment.

In a second example, large swatches may be etched onto the lens whileavoiding small “dot” areas that correspond to the dot pattern. When theIR laser emits a beam of IR light through this type of lens, only IRlight that passes through the small unetched “dot” areas will passunimpededly, thus projecting a dot pattern into the surroundingenvironment. Any other technique for generating a dot pattern may alsobe used (e.g., instead of etching the lens, a dot-pattern covering maybe placed on the lens). Additionally, any other DOE may be used todisperse IR light in accordance with a pre-configured dot pattern.Regardless of its implementation, a beam of IR light is dispersedaccording to a predetermined dot-pattern.

While the disclosure has used the phrase “dot pattern,” it will beappreciated that the term “dot” does not limit the illuminations to acircular shape. In fact, any shape of dot may be projected in thepredetermined dot-pattern. For example, the dot pattern may include apattern of circles, triangles, squares, rectangles and/or any otherpolygon or oval shaped dot(s).

It will also be appreciated that any kind of dot “pattern” may be used.For instance, the pattern may be a completely random assortment of dots.Alternatively, the pattern may include a pre-configured pattern thatrepeats itself in a horizontal and/or vertical direction. By repeatingthe dot pattern at least in the vertical direction, advantages arerealized because the efficiency of the stereo matching algorithm will beimproved. Even further, the size of the dots may also vary such thatsome dots may be larger or smaller than other dots to facilitate patternmatching.

As described herein, the IR dot pattern illumination 410 is projectedinto the surrounding environment of a HMD 403 in order to project, orrather add, “texture” to object surfaces in the surrounding environment400. In implementations where the HMD 403 includes a stereo camerasystem, the overlap of the IR dot pattern illumination 410 with thefield of view region of the stereo camera system enables both of the HMD403's cameras to detect at least a part of the IR dot patternillumination 410 being reflected off of objects in the overlapping fieldof view region. In this manner, the cameras are able to obtain digitalimages that include digital content corresponding to the texture (i.e.the “obtained” texture is actually reflected IR light generated as aresult of the IR dot pattern illumination 410 reflecting off of surfacesin the environment 400). Using the digital content corresponding to thetexture, the HMD 403 is able to compute pixel disparity, thusdetermining depth of objects (even smooth objects) in the overlappingfield of view region. In some instances, because the stereo cameras 205and 210 are sensitive to both IR light and visible light, the digitalimages obtained by the cameras include IR dot pattern illuminationreflections (for smooth objects) in addition to readily identifiablefeatures (for textured objects) which do not require IR illumination fordepth calculation.

Additionally, or alternatively, in some embodiments the one or morecameras of the HMD 403 may utilize the reflections of the IR dot patternillumination 410 light as anchor points to make tracking of the HMD morerobust. This is particularly beneficial in low-light environments, asdescribed hereinbelow.

Attention is now directed to FIG. 5. This FIG. 5 illustrates an IRprojection pattern illuminator 505 configured to project an IR dotpattern illumination 510 that selectively illuminate portions of anenvironment 500 surrounding an HMD 503 based on certain criteria. Asillustrated, the IR dot pattern illumination 510 projected by IRprojection pattern illuminator 505 is substantially rectangular inshape. This arrangement is illustrative only, and it will be appreciatedthat an IR projection pattern illuminator 505 may be configured toproject an IR illumination in a variety of shapes and manners. By way ofexample, the IR projection pattern illuminator 505 may be configured toproject a

$\frac{\pi}{2}$steradian IR dot pattern illumination to illuminate a circular region.

In some embodiments, the IR projection pattern illuminator 505 receivesand/or generates (e.g., via a processor based on data received)instructions which dictate the manner in which the IR projection patternilluminator 505 will emit an IR dot pattern illumination 510. The datareceived may arise from a communication (wireless or wired) between theHMD 503 and the IR projection pattern illuminator 505, from otherexternal sensors, from sensors attached to the IR projection patternilluminator, or from other sensors or sources, as will be discussed inanother part of this disclosure.

As mentioned, the IR projection pattern illuminator 505 may utilize thereceived and/or generated instructions from the HMD or a third partysystem to determine the characteristics of the IR dot patternillumination 510 to be projected. The characteristics of the IR dotpattern illumination 510 to be projected may be based on a variety ofcriteria. For example, as illustrated in FIG. 5, the IR projectionpattern illuminator 505 may wirelessly receive data indicating motiondata for the HMD 503. For example, the data may indicate that the HMD503 is being turned to the left (according to the illustrated arrowproximate the HMD 503), as illustrated, which consequently indicatesthat the fields of view of any cameras attached to the HMD 503 will alsoturn to the left. In response to the received data, the IR projectionpattern illuminator 505 adjusts the region upon which the IR dot patternillumination 510 is projected to correspond with the changing field ofview of any cameras attached to the HMD 503.

In addition to adjusting the direction and/or size of the region uponwhich the IR dot pattern illumination 510 is projected, the IRprojection pattern illuminator 505 is, in some embodiments, configuredto adjust the illumination strength of the IR dot pattern illumination510 (e.g., the intensity of the emitted light). In other embodiments,data indicating which particular surfaces in the environment 500 areperceived as lacking in features with which to perform active stereodepth computations (e.g., flat) is utilized by the IR projection patternilluminator 505 to selectively illuminate only those portions of theenvironment 500 and/or increase a density of textured dot pattern lightprojected to those portions that lack necessary features for performingactive stereo depth computation with the HMD.

In yet another embodiment, the IR projection pattern illuminator 505senses that the ambient lighting in the environment 500 (or, forexample, the number of detectable anchor points) has reduced below apredetermined threshold. In response to this detection, the IRprojection pattern illuminator 505 increases the illumination strength(e.g., intensity of emitted light), and/or enters an IR flood projectionmode, as described in more detail below.

In another embodiment, the HMD determines that an efficiency ofperforming stereo depth computations is dropping below a thresholdvalue, due to a lack of detectable/textured features in the surroundingenvironment (based on analyzing images obtained with the HMD camerasystem). In response to this determination, the HMD communicatesinstructions to the projector to modify the illumination pattern, bychanging at least one of: a direction for projecting the illuminationpattern, an intensity of the illumination pattern or a density orconfiguration of the illumination pattern.

The exemplary criteria herein described merely illustrate a few of thepossible criteria upon which the IR projection pattern illuminator 505may base alterations to the IR pattern projection being emitted thereby.Other criteria exist which are not specifically addressed in thisdisclosure (e.g., the capture rate of one or more sensors in thesystem).

The disclosure has focused on structured projection patterns includingdot patterns and variants thereon. The structured projection patternsdescribed herein may also be varied as to their fill rate, or thepercentage of space in the IR projection region that will beilluminated. IR projection pattern illuminators may be designed with afill rate of 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or 100%. Afill rate of 100% corresponds to a flood of IR light, which isparticularly advantageous in circumstances with low ambient light.

A discussion on ambient light is pertinent. For reference, a brightsunny day typically has an ambient light intensity of around10,000-50,000 lux. An overcast day typically has an ambient lightintensity of around 1,000-10,000 lux. An indoor office typically has anambient light intensity of around 100-300 lux. The time of daycorresponding to twilight typically has an ambient light intensity ofaround 10 lux. Deep twilight has an ambient light intensity of around 1lux. As used herein, the illustrated low light environment 1300 leastcorresponds to any environment in which the ambient light intensity isat or below about 40 lux.

In certain embodiments, an IR projection pattern illuminator relies onone or more sensors to determine whether a low-light environment exists.These sensors may be located, for example, on the HMD, on the IRprojection pattern illuminator itself, or at any location in theenvironment.

FIG. 6 illustrates an IR projection pattern illuminator 605 emitting aflood of IR light 610 into a low-light environment 600 surrounding a HMD603. As used herein, the IR projection pattern illuminator 605 includesone or more of an IR LED, an array of IR LEDs, an IR laser diode, anincandescent discharge illuminator, a VCSEL, a plasma dischargeilluminator, etc. The IR projection pattern illuminator 605 alsoincludes, in some embodiments, a corresponding collimating optic. The IRilluminator is configured to emit a broad-beamed, homogeneous ray oflight that spans a large illumination area. By “homogeneous,” it ismeant that all illumination points are subjected to substantially thesame amount of light.

In this manner, the flood IR illuminator 1305 is configured to emit aflood of IR light spread across a selected field of view that at leastpartially illuminates the head tracking camera's FOV. This field of viewmay be spanned differently. For example, the field of view may span 90degrees, 100 degrees, 110 degrees, 120 degrees, 130 degrees, and so on.Additionally, the flood IR illuminator 1305 may be but one of an arrayof multiple illuminators.

In some circumstances, the IR projection pattern illuminator 605 alwaysemit a flood of IR light 710 while the HMD 703 is actively generating amixed-reality environment. In other circumstances, the IR projectionpattern illuminator 605 is triggered to turn on only when a particularcondition occurs.

Examples of triggering conditions include, but are not limited to, (1) adetection (either by the HMD 703's camera(s) or by an ambient lightsensor of the HMD) that the ambient light intensity of the surroundingenvironment has fallen below a threshold intensity value (e.g., 40 lux)and/or (2) a determination that the number of detectable/detected anchorpoints is below a preselected number of anchor points (e.g., 6, 7, 8, 9,10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 90, 100, 200, etc. anchorpoints).

As shown in FIG. 6, the IR projection pattern illuminator 605 isemitting a flood of IR light 610 into the environment 600. Such aprocess is particularly beneficial for helping to illuminate potentialanchor points that may be used to track movements and to therebyfacilitate identification of the anchor points by the HMD when sensingreflected light from the anchor points.

Such a scenario is shown in FIG. 7 (the illustration of the flood of IRlight 710 is relaxed compared to the illustration of the flood of IRlight 610 in FIG. 6, so as to better draw attention to the anchor points715 herein described). It will be appreciated that the camera(s) of HMD703 are able to detect both visible light and IR light. In someinstances, the cameras capture images of reflected IR light and usethese IR light images to detect anchor points in any kind ofenvironment, even in low light environments. It will be appreciated thatthe camera system of the HMD may include any number of cameras (e.g., 1,2 or more cameras) that are able to (collectively or individually) senseonly visible light, only IR light or a combination of both IR light andvisible light.

As shown in FIG. 7, the HMD 703 is able to detect a vast array ofdifferent anchor points 715 by sensing reflected IR light and/or visiblelight. These anchor points 715 are symbolically identified through theuse of the dark circles. In some scenarios, these anchor points arefixed physical features, such as corners, that are useful fordetermining the HMD 703's position and orientation in relation to itssurrounding environment. It will be appreciated that, in a low-lightenvironment, the anchor points 715 are made more readily ascertainableto the cameras on the HMD 703 by the flood of IR light 710 projected bythe IR projection pattern illuminator 705. Accordingly, with a greaternumber of ascertainable anchor points 715, the HMD 703 is able toutilize the anchor points 715 as a reference for tracking the movementof the HMD 703, even in low-light environments.

The disclosure thus far has focused on embodiments including one IRprojection pattern illuminator. It will be appreciated, however, thatimplementations including more than one IR projection patternilluminator are within the scope of this disclosure. It will also beappreciated that, in some embodiments, the IR projection patternilluminator is configured to additionally or alternatively projectvisible flood light and/or structured light patterns (e.g., a dotpattern).

Attention is now directed towards FIG. 8, which illustrates two IRprojection pattern illuminators 805 projecting an IR dot patternillumination 810 onto an environment 800 surrounding a HMD 803. Asillustrated, the dot pattern illumination is projected in part by afirst IR projection pattern illuminator 805 in one position (depicted inthe lower-left portion of FIG. 8) and in part by a second IR projectionpattern illuminator 805 in another position (depicted in the upper-rightportion of FIG. 8).

In some embodiments, the IR projection pattern illuminators 805 areconfigured to illuminate only a portion of the environment 800.Alternatively, in other embodiments, both of the IR projection patternilluminators 805 are able to illuminate the entire environment 800,thereby overlapping their projected illumination areas. Each of the IRprojection pattern illuminators 805 are also able, in some embodiments,to selectively illuminate limited portions of the environment 800 basedon any predetermined criteria (e.g., HMD pose data, detectedenvironmental shadowing, lack of detected anchor points, user attentiondirected to a particular area of the environment, etc.), regardless ofwhether the projected illuminations are overlapping or not.

Additionally, or alternatively, the IR projection pattern illuminators805 may also utilize criteria which enable the IR projection patternilluminators 805 to ensure that the IR dot pattern illumination lightreaches the desired object(s) in the environment 800 surrounding the HMD803. When a HMD user's body or other object stands between an IRprojection pattern illuminator 805 and one or more objects in anenvironment 800, the IR dot pattern illumination light may be occludedor otherwise impeded from reaching objects to be mapped. Accordingly, insuch a scenario, the one or more cameras attached to the HMD 803 may beunable to detect any “texture” that would have been added by the IR dotpattern illumination to the object(s).

For example, turning to FIG. 8, the user of HMD 803 may be positionedbetween the lower-left IR projection pattern illuminator 805 such thatlight from that illuminator 805 does not reach the table depicted.Without an additional illuminator, any cameras attached to the HMD 803would be unable to detect any “texture” that would have been added bythe IR dot pattern illumination to the table, and thus the HMD 803 wouldbe unable to determine a depth map for flat surfaces of the table.

To solve this problem, in some embodiments, two or more IR projectionpattern illuminators 805 are utilized such that even when a user's orother object's position would obstruct one IR projection patternilluminator, the second will compensate for the obstruction by providingthe IR dot pattern illumination from a different location/projector.Returning to the aforementioned example in FIG. 8, when the HMD 803user's body prevents light from the lower-left IR projection patternilluminator 805 from reaching the table, the upper-right IR projectionpattern illuminator 805 is configured to illuminate the table.Accordingly, a stereo camera pair attached to HMD 803 is still able todetect additional “texture” on the table and determine a depth map forflat surfaces of the table according to the present disclosure.

It will be appreciated that IR projection pattern illuminators 805 maybe configured to communicate with the HMD 803, or with each other, tocoordinate and/or alter their IR dot pattern illumination projections810, as described hereinabove. For example, the HMD 803 may transmitpose data to the IR projection pattern illuminators 805, which willcause the illuminators to illuminate portions of the environment 800that correspond to the fields of view of the stereo cameras of the HMD803. In some instances, this will involve only one IR projection patternilluminator 805 becoming activated. In other instances, both IRprojection pattern illuminators will become activated. In yet otherinstances, one IR projection pattern illuminator will project IR lightwith a different intensity than the intensity used by the secondilluminator.

As briefly introduced earlier, any of the IR dot-pattern illuminatorsand/or the flood IR illuminators may be synchronized with the exposureperiods of one, some, or all of the HMD's cameras. In this manner, theIR dot-pattern illuminators and/or the flood IR illuminators may operatein a pulsed or staggered mode. While in this mode, the IR illuminators(e.g., the IR dot-pattern illuminators and/or the flood IR illuminators)are able to selectively emit IR light during times corresponding to thecameras' exposure periods and not emit light during times when thecameras' are not sensing light. Such configurations are beneficialbecause they help preserve or prolong the HMD's battery lifespan.Accordingly, any of the IR illuminators may be time synchronized withany of the HMD's cameras.

Attention is now directed to FIG. 9, which illustrate various examplecomputer system components that may be utilized to improve depthdetection with stereo tracking cameras and/or head tracking. Theillustrated HMD computer system components 901 may be integrated into aHMD. For instance, these components may be part of the HMD 200 that wasdiscussed in connection with FIG. 2 or any of the other HMDs discussedthus far.

As shown, the HMD computer system components 901 include camera(s) 905which may be an example implementation of the left camera 205 and theright camera 210 from FIG. 2 or any of the other cameras discussedherein.

The HMD computer system components 901 also include a head tracker 915and a depth detector 930, as well as one or more processors 935 andstorage 940. The one or more processors 935 include hardware processors,each of which may be a specially configured processor (e.g., a graphicsprocessing unit (GPU) or as an application specific integrated circuit(“ASIC”)).

In some instances, the processor(s) 935 include one processor that isconfigured to operate as the head tracker 915 and an additionalprocessor that is configured to operate as the depth detector 930.Alternatively, a single processor is used to operate as both the headtracker 915 and the depth detector 930. More details on processors andhow they operate will be provided in connection with FIG. 12.Additionally, the head tracker 915 and the depth detector 930 will bedescribed in more detail in conjunction with the methods illustrated inFIGS. 10 and 11. The processor(s) 935 also execute storedcomputer-executable instructions stored in the storage 940 forimplementing the acts described herein.

FIG. 9 also illustrates various projector computer system components903, including one or more IR dot pattern illuminators 925 and one ormore IR flood illuminators 910, both of which are exampleimplementations of the IR projection pattern illuminator 405 from FIG. 4or any of the other IR projection pattern illuminators discussed herein.

Additionally, the projector computer system components also include oneor more processors 935 and storage 940. In some instances, theprocessor(s) 935 include one processor that is configured to operate IRdot pattern illumination and an additional processor that is configuredto operate IR flood illumination. Alternatively, a single processor isused to operate both IR dot pattern and IR flood illumination.Furthermore, the processor(s) 935 are, in some instances, configured toadjust the IR projection pattern illumination according to certaincriteria (e.g., pose of the HMD).

Both the HMD computer system components 901 and projector computersystem components 903 include a wireless data transmitter 950. Thisenables data communication between the HMD computer system components901 and the projector computer system components 903 (as denoted by thedashed line between the HMD computer components 901 and the projectorcomputer components 903), even for configurations in which the HMDcomputer system/components 901 are untethered from (e.g., not wired to)the projector(s)/components 903. Accordingly, data may be transferredbetween the two computer systems to fulfill the objectives of theembodiments herein described (e.g., dynamic adjustment of the IRprojection pattern illumination). More details on the operation of thewireless data transmitters 950 will be provided in connection with FIG.12.

The following discussion now refers to a number of methods and methodacts that may be performed. Although the method acts may be discussed ina certain order or illustrated in a flow chart as occurring in aparticular order, no particular ordering is required unless specificallystated, or required because an act is dependent on another act beingcompleted prior to the act being performed.

As shown in FIG. 10, a flowchart 1000 is provided with various actsassociated with methods for adding predetermined texture to objectslocated within an environment of a HMD. The first illustrated act is anact for projecting an IR dot pattern illumination from projector that isphysically untethered from a HMD (act 1001), such as a projector that ismounted in the environment surrounding the HMD. For reference, this actmay be performed by the projector 903 from FIG. 9.

In some embodiments, the HMD initially uses its cameras to detect anamount of reflected IR light generated as a result of the IR projectionpattern illuminator emitting the IR dot-pattern illumination and/or anamount of reflected visible light. In response to a detected “intensity”of the reflected IR light and/or ambient light, which intensity may bedetected using the cameras, the HMD modifies a power level of the IRprojection pattern illuminator until a predetermined threshold intensityof the reflected IR light is detected by the cameras. As such, the HMDis able to dynamically modify/adjust the power level of the IRprojection pattern illuminator in order to increase or decrease theintensity of the IR light, when necessary (e.g., in low lightconditions) to obtain a desired level of detail in the captured imagesused for depth detection and/or tracking. Such adjustments areparticularly beneficial for enabling the HMD to use only as much poweras necessary in obtaining sufficiently adequate images for determiningdepth. The modifications may also include modifying a patternconfiguration and/or projection area for the illumination pattern and/ora type of illumination (e.g., quantity of IR light vs. visible lightbeing emitted).

Returning to FIG. 10, reflected IR light and/or reflected visible lightis subsequently detected (act 1003). This act is performed by thecameras 205 and 210 described in FIG. 2 and throughout this paper.Notably, this reflected IR light is generated as a result of at least aportion of the IR dot-pattern illumination reflecting off of the surfacein the environment (e.g., the smooth table top from FIG. 4).

To detect this reflected IR light and/or reflected visible light, afirst image, which includes digital content corresponding to a firstpart of the IR dot-pattern illumination and/or the reflected visiblelight, is captured using one of the cameras. Concurrently with thatoperation, a second image is also captured using the second camera. Thesecond image includes digital content corresponding to a second part ofthe IR dot-pattern illumination and/or the reflected visible light andwhich includes digital content corresponding to at least some of thesame IR dot-pattern illumination and/or visible light that was capturedin the first image.

The first image and the second image are then used to determine a depthfor at least the common part of the illumination area that was capturedin both images (act 1005). By obtaining images of the reflected IR lightand/or reflected visible light using both a left camera and a rightcamera, the HMD is able to measure the pixel disparity present betweencommon pixels in the two images. This pixel disparity may then be usedby the depth detector 930 of FIG. 9 to determine the stereoscopic depthfor that object.

Notably, HMD is able to determine a depth for objects in theenvironment, even when those objects have smooth surfaces because of theIR dot-pattern that adds texture to the objects. By determining thedepth for objects in this manner (i.e. stereo vision), it is notnecessary to utilize additional hardware/components to performtime-of-flight computations. For reference, time-of-flight systems aremuch more expensive because they use additional hardware components. Incontrast, the current embodiments re-purpose many existing components sothat they can perform new or additional functionalities, thus savingsignificant costs and thereby reducing computational and power burdenson the HMDs.

It will be appreciated that the application and use of an IR projectionpattern illuminator can significantly improve the manner in which depthdetection occurs for HMDs, and while reducing costs of the HMDs (e.g.,by reducing resource competition and resource allocation issues). Suchprocesses are also particularly beneficial when the environmentsurrounding the HMD includes smooth surfaced objects.

FIG. 11 illustrates a flowchart 1100 of various acts associated withmethods for tracking movements of a HMD in any kind of environment,including low light environments. Similar to the methods described inreference to FIG. 10, the methods of FIG. 11 may be performed by the HMDcomputer system components 901 and the projector computer systemcomponents 903 shown in FIG. 9.

The first illustrated act is an act of emitting a flood of light from aprojector that is physically untethered from the HMD (act 1101). Thisact may be performed by the projector 903 from FIG. 9, for example. Thislight may include any combination of IR light and/or visible light.

Next, reflected light is detected (act 1103). This reflected light mayinclude any combination of reflected IR pattern light and visible light.The light is sensed by the HMD camera system, such as camera(s) 905 fromFIG. 9. In some embodiments, the light is detected by a HMD'shead-tracking stereo camera pair. Based on the detected light, aposition of the HMD is determined (act 1105). This act may be performedby the head tracker 915 of FIG. 9, for example. Detecting the positionof the HMD may be performed in the manner described above in referenceto the inside-out camera detection techniques.

Therefore, by incorporating the use of one or more illuminators,including one or more IR pattern illuminator(s) and/or visible lightilluminator(s), the HMD is able to better track movements within anykind of environment, including low light environments.

It will further be appreciated that the HMD is capable of recognizing anIR dot pattern illumination and other visible light reflections as aplurality of anchor points fixed to elements of the environmentsurrounding the HMD. Accordingly, those skilled in the art willrecognize that projecting an IR dot pattern illumination with an IRprojection pattern illuminator in an environment surrounding a HMD mayaugment other tracking processes performed by the HMD for simultaneouslyallowing for active stereo depth computations with as well as improvedhead tracking by the HMD.

Example Computer System

Having just described the various features and functionalities of someof the disclosed embodiments, the focus will now be directed to FIG. 12which illustrates an example computer system 1200 that may be used tofacilitate the operations described herein. In particular, this computersystem 1200 may be in the form of the HMDs or the projectors that weredescribed earlier.

In fact, the computer system 1200 may take various different forms. Forexample, in FIG. 12, the computer system 1200 is embodied as a HMD.Although the computer system 1200 may be embodied as a HMD, the computersystem 1200 may also be a distributed system that includes one or moreconnected computing components/devices that are in communication withthe HMD. Accordingly, the computer system 1200 may be embodied in anyform and is not limited strictly to the depiction illustrated in FIG.12. By way of example, the computer system 1200 may include a projector,desktop computer, a laptop, a tablet, a mobile phone, server, datacenter and/or any other computer system.

In its most basic configuration, the computer system 1200 includesvarious different components. For example, FIG. 12 shows that computersystem 1200 includes at least one hardware processing unit 1205 (aka a“processor”), input/output (I/O) interfaces 1210, graphics renderingengines 1215, one or more sensors 1220, and storage 1225. More detail onthe hardware processing unit 1205 will be presented momentarily.

The storage 1225 may be physical system memory, which may be volatile,non-volatile, or some combination of the two. The term “memory” may alsobe used herein to refer to non-volatile mass storage such as physicalstorage media. If the computer system 1200 is distributed, theprocessing, memory, and/or storage capability may be distributed aswell. As used herein, the term “executable module,” “executablecomponent,” or even “component” can refer to software objects, routines,or methods that may be executed on the computer system 1200. Thedifferent components, modules, engines, and services described hereinmay be implemented as objects or processors that execute on the computersystem 1200 (e.g. as separate threads).

The disclosed embodiments may comprise or utilize a special-purpose orgeneral-purpose computer including computer hardware, such as, forexample, one or more processors (such the hardware processing unit 1205)and system memory (such as storage 1225), as discussed in greater detailbelow. Embodiments also include physical and other computer-readablemedia for carrying or storing computer-executable instructions and/ordata structures. Such computer-readable media can be any available mediathat can be accessed by a general-purpose or special-purpose computersystem. Computer-readable media that store computer-executableinstructions in the form of data are physical computer storage media.Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example and not limitation, thecurrent embodiments can comprise at least two distinctly different kindsof computer-readable media: computer storage media and transmissionmedia.

Computer storage media are hardware storage devices, such as RAM, ROM,EEPROM, CD-ROM, solid state drives (SSDs) that are based on RAM, Flashmemory, phase-change memory (PCM), or other types of memory, or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium that can be used to store desired programcode means in the form of computer-executable instructions, data, ordata structures and that can be accessed by a general-purpose orspecial-purpose computer.

The computer system 1200 may also be connected (via a wired or wirelessconnection) to external sensors 1230 (e.g., one or more remote cameras,accelerometers, gyroscopes, acoustic sensors, magnetometers, etc.). Itwill be appreciated that the external sensors include sensor systems(e.g., a sensor system including a light emitter and camera), ratherthan solely individual sensor apparatuses. Further, the computer system1200 may also be connected through one or more wired or wirelessnetworks 1235 to remote systems(s) 1240 that are configured to performany of the processing described with regard to computer system 1200.

During use, a user of the computer system 1200 is able to perceiveinformation (e.g., a mixed-reality environment) through a display screenthat is included among the I/O interface(s) 1210 and that is visible tothe user. The I/O interface(s) 1210 and sensors 1220/1230 also includegesture detection devices, eye trackers, and/or other movement detectingcomponents (e.g., cameras, gyroscopes, accelerometers, magnetometers,acoustic sensors, global positioning systems (“GPS”), etc.) that areable to detect positioning and movement of one or more real-worldobjects, such as a user's hand, a stylus, and/or any other object(s)that the user may interact with while being immersed in the scene.

The graphics rendering engine 1215 is configured, with the hardwareprocessing unit 1205, to render one or more virtual objects within thescene. As a result, the virtual objects accurately move in response to amovement of the user and/or in response to user input as the userinteracts within the virtual scene.

A “network,” like the network 1235 shown in FIG. 12, is defined as oneor more data links and/or data switches that enable the transport ofelectronic data between computer systems, modules, and/or otherelectronic devices. When information is transferred, or provided, over anetwork (either hardwired, wireless, or a combination of hardwired andwireless) to a computer, the computer properly views the connection as atransmission medium. The computer system 1200 will include one or morecommunication channels that are used to communicate with the network1235. Transmissions media include a network that can be used to carrydata or desired program code means in the form of computer-executableinstructions or in the form of data structures. Further, thesecomputer-executable instructions can be accessed by a general-purpose orspecial-purpose computer. Combinations of the above should also beincluded within the scope of computer-readable media.

Upon reaching various computer system components, program code means inthe form of computer-executable instructions or data structures can betransferred automatically from transmission media to computer storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface module (e.g., a network interface card or“NIC”) and then eventually transferred to computer system RAM and/or toless volatile computer storage media at a computer system. Thus, itshould be understood that computer storage media can be included incomputer system components that also (or even primarily) utilizetransmission media.

Computer-executable (or computer-interpretable) instructions comprise,for example, instructions that cause a general-purpose computer,special-purpose computer, or special-purpose processing device toperform a certain function or group of functions. Thecomputer-executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the embodiments may bepracticed in network computing environments with many types of computersystem configurations, including personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, and the like. The embodiments may alsobe practiced in distributed system environments where local and remotecomputer systems that are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network each perform tasks (e.g. cloud computing, cloudservices and the like). In a distributed system environment, programmodules may be located in both local and remote memory storage devices.

Additionally or alternatively, the functionality described herein can beperformed, at least in part, by one or more hardware logic components(e.g., the hardware processing unit 1205). For example, and withoutlimitation, illustrative types of hardware logic components that can beused include Field-Programmable Gate Arrays (FPGAs), Program-Specific orApplication-Specific Integrated Circuits (ASICs), Program-SpecificStandard Products (ASSPs), System-On-A-Chip Systems (SOCs), ComplexProgrammable Logic Devices (CPLDs), Central Processing Units (CPUs), andother types of programmable hardware.

The disclosed embodiments provide various advantages over traditionalHMD systems. Some of these advantages include providing a more robustand accurate depth determination for mixed-reality environments,particularly low light environments. Additionally, some of theseadvantages include the ability to track movement (e.g., head movement,hand movement, etc.) in any kind of environment, even low lightenvironments. Furthermore, by repurposing existing hardware components,such as the head tracking cameras to additionally perform depthdetection, the disclosed embodiments can reduce/simplify the costs,power consumption and form factor of the HMD systems.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or characteristics. The described embodimentsare to be considered in all respects only as illustrative and notrestrictive. The scope of the invention is, therefore, indicated by theappended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency of the claims areto be embraced within their scope.

What is claimed is:
 1. A system for performing active stereo depthcomputation of a head-mounted display (HMD) in a surroundingenvironment, the system comprising: a projector comprising one or moredot pattern illuminators for projecting a dot pattern to the surroundingenvironment and that is at least partially reflected in the surroundingenvironment as reflected light, the projector being triggered to projectthe dot pattern in response to one or more triggering conditions, amongwhich include a detected amount of ambient light or a detected number ofanchor points being at a certain predetermined value; and the HMD,wherein the HMD is physically untethered from the projector, the HMDcomprising: a first camera and a second camera having an overlappingfield of view; and one or more processors and one or morecomputer-readable hardware storage devices having stored thereoncomputer-executable instructions that are executable by the one or moreprocessors to cause the HMD to compute a depth of one or more objectswithin the overlapping field of view relative to the HMD and based atleast in part on the reflected light from the dot pattern that isdetected by both the first camera and the second camera, whereintriggering the projector to project the dot pattern, which is relied onto compute the depth of the one or more objects, is performed inresponse to determining that the detected number of anchor points, whichare identified within the surrounding environment using the first cameraand the second camera, is below a preselected number of anchor points.2. The system of claim 1, wherein the system further includes one ormore flood illuminators for projecting a flood of light to thesurrounding environment and that is at least partially reflected from inthe surrounding environment as reflected flood light detected by atleast one of the first and second camera and that is used to track themovement of the HMD in the surrounding environment.
 3. The system ofclaim 1, wherein the dot pattern is a pre-configured pattern thatrepeats itself in at least a horizontal direction.
 4. The system ofclaim 1, wherein the projector includes one or more flood illuminatorsthat project a flood of light to the surrounding environment.
 5. Thesystem of claim 1, wherein the one or more dot pattern illuminators arepulsed to synchronize with an exposure of at least one of the firstcamera or the second camera.
 6. A system for performing tracking of ahead-mounted display (HMD) in a surrounding environment, the systemcomprising: a projector comprising one or more infrared light dotpattern illuminators for projecting an infrared (IR) light dot patternto the surrounding environment and that is at least partially reflectedin the surrounding environment as reflected IR light, the projectorbeing triggered to project the IR light dot pattern in response to oneor more triggering conditions, among which include a detected amount ofambient light or a detected number of anchor points being at a certainpredetermined value; and the HMD, wherein the HMD is physicallyuntethered from the projector, the HMD comprising: a camera systemhaving at least one camera sensor for detecting the reflected IR light;and one or more processors and one or more computer-readable hardwarestorage devices having stored thereon computer-executable instructionsthat are executable by the one or more processors to cause the HMD totrack movement of the HMD based at least in part on the reflected IRlight that is detected by the at least one camera sensor, whereintriggering the projector to project the IR light dot pattern, which isrelied on to track the movement of the HMD, is performed in response todetermining that the detected number of anchor points, which areidentified within the surrounding environment using the camera system,is below a preselected number of anchor points.
 7. The system of claim6, wherein the camera system comprises a first camera and a secondcamera having an overlapping field of view and wherein the first cameraand the second camera obtain corresponding images of the reflected IRlight to perform depth detection of at least one object in thesurrounding environment.
 8. The system of claim 6, wherein the IR lightdot pattern is a pre-configured pattern that repeats itself in at leasta horizontal direction.
 9. The system of claim 6, wherein the projectorincludes one or more IR flood illuminators that project a flood of IRlight to the surrounding environment.
 10. The system of claim 6, whereinthe one or more infrared light dot pattern illuminators include at leasta first illuminator and at least a second illuminator, the firstilluminator projecting IR light with a different intensity than anintensity used by the second illuminator.
 11. A method for performing atleast one of depth detection and tracking with a head-mounted display(HMD) in an environment that includes a projector that is untetheredfrom the HMD, the method comprising: causing the projector to project aninfrared (IR) dot pattern onto objects located within the environment,the projector being triggered to project the IR light dot pattern inresponse to one or more triggering conditions, among which include adetected amount of ambient light or a detected number of anchor pointsbeing at a certain predetermined value; detecting at least a portion ofthe IR dot pattern with one or more cameras attached to the HMD; andutilizing the at least the portion of the IR dot pattern detected by theone or more cameras to perform at least one of depth detection andtracking with the HMD, wherein triggering the projector to project theIR light dot pattern, which is relied on to either perform the depthdetection or to perform the tracking with the HMD, is performed inresponse to determining that the detected number of anchor points, whichare identified within the environment using the one or more cameras, isbelow a preselected number of anchor points.
 12. The method of claim 11,wherein the method includes utilizing the at least the portion of the IRdot pattern detected by the one or more cameras to perform the trackingwith the HMD.
 13. The method of claim 11, wherein the one or morecameras include a first camera and a second camera, and wherein themethod includes utilizing the at least the portion of the IR dot patterndetected by the first and second cameras to perform the depth detectionwith the HMD.
 14. The system of claim 1, wherein the one or moretriggering conditions further includes determining that an ambient lightintensity of the surrounding environment is below a threshold intensityvalue such that triggering the projector to project the dot pattern isperformed in response to the ambient light intensity being below thethreshold intensity value.